Image capture with privacy protection

ABSTRACT

Surveillance systems process video streams obtained by a surveillance device, such as a drone, to either obscure or highlight objects in a surveillance area based on tags associated with the objects. A method of providing obscurant data includes receiving image data including an image of a target and receiving a preference setting corresponding to the target. Obscurant data of at least a portion of the image data corresponding to the target are determined using the received preference setting. A method of providing surveillance image data includes capturing image data including an image of a target, querying a database to receive a preference setting corresponding to the target, determining the obscurant data of the portion of the image data, and selectively modifying the received image data according to the determined obscurant data to provide the surveillance image data.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is a continuation of U.S. patent application Ser. No.15/948,927, filed Apr. 9, 2018, which is a continuation of U.S. patentapplication Ser. No. 15/298,946, filed Oct. 20, 2016, now U.S. Pat. No.9,940,525, which is a continuation-in-part of U.S. patent applicationSer. No. 14/084,071, filed Nov. 19, 2013, which claims the benefit ofU.S. Provisional Patent Application Ser. Nos. 61/727,899, filed Nov. 19,2012, and 61/774,722, filed Mar. 8, 2013, the entirety of each of whichis incorporated herein by reference.

FIELD OF THE DISCLOSURE

The present application relates to image capture systems and methods,and more specifically to those responsive to inputs to selectively maskportions of their outputs.

BACKGROUND OF THE DISCLOSURE

Remote surveillance is common for security, law enforcement,intelligence, and military purposes. For example, shopping centers,police stations, and other facilities catering to the public often haveclosed-circuit television (CCTV) cameras or other devices monitoringmembers of the public in the facility. In another example, pole-mountedcameras have been used for traffic enforcement. One type of surveillancethat is on the rise is drone surveillance. Unmanned aerial vehicles(UAVs, also known as “drones”) are aircraft that do not carry humanpilots or passengers, and are commonly used for surveillance and combat.Drones can be configured as fixed wing aircraft, helicopters, or otheraerial vehicles. A human pilot generally controls a drone using awireless link from one or more ground stations, but drones can includeautonomous systems that perform the functions normally executed by apilot. Drones serve to carry sensors and permit those sensors tointeract with their environments in order to collect data. Drones can beused, for example, to carry out surveillance or intelligence-gatheringmissions using a variety of optical or other sensors, to transport goodsor passengers, or to locate and respond to threats.

Various software exists for detecting moving objects or recognizingfaces in captured image data, and then displaying an indicator of thedetection. Various software also exists for tracking such detected imagefeatures as they move. Moreover, various techniques are used to improvethe visibility of objects in captured image data. For example, militaryforces wear IR beacons in combat areas. These IR beacons blink ininfrared wavelengths. This renders friendly forces visible in an IRcamera view, reducing the probability of casualties due to friendlyfire.

Drones can range in size from, e.g., small units weighing grams, toairplanes with wingspans over ten feet, to full-sized airplanes such asbombers. Particularly with smaller units, drones can permit covertsurveillance of persons in public. For example, a drone airplane with awingspan of approximately JO', painted appropriately, with an electricdrive, can be effectively invisible and inaudible to persons on theground at an altitude of as little as 500′ above ground level (AGL).There is, therefore, a need of implementing the rights of people to befree from surveillance without cause, e.g., privacy and due-processrights, especially when the surveillance is being carried out by asmall, unobtrusive drone. This need can also pertain to surveillancecarried out by fixed cameras such as security or traffic cameras.

SUMMARY OF THE DISCLOSURE

According to various aspects, there is provided a method of providingobscurant data, comprising: receiving image data including an image of atarget; receiving a preference setting corresponding to the target; anddetermining the obscurant data of at least a portion of the image datacorresponding to the target using the received preference setting.

Optionally, the obscurant data can be determined further using thereceived image data. The at least a portion can correspond to the imageof the target. The method can include selectively modifying the receivedimage data according to the determined obscurant data to provide asurveillance image. The determining step can include determining atleast a portion of the image data to be obscured in response to apreference setting requesting privacy and the modifying step includesobscuring the determined at least a portion of the image data. Thedetermining step can include determining at least a portion of the imageto be indicated in response to a preference setting requesting increasedvisibility of the target and the modifying step can include modifyingthe determined at least a portion of the image data to include data of avisible indicator. The method can further include receiving a uniqueidentifier of the target. The step of receiving a preference setting caninclude transmitting the received unique identifier to a database andreceiving from the database the preference setting corresponding to theunique identifier. The image data can correspond to a visibility region,and the step of receiving a preference setting step can includetransmitting data of the visibility region to a database to determinewhether the target is present in the visibility region; and receivingfrom the database the preference setting or an indication that thetarget is not present in the visibility region. The method can furtherinclude transmitting to a database a timestamp corresponding to theimage data. The method can further include storing the determinedobscurant data in a storage device. The storage device can be atamper-evident storage device. The target can be selected from the groupconsisting of a person, a building, a vehicle or an animal, thedetermining step can include determining a size of the at least aportion of the image data using the received image data. The method canfurther comprise receiving a validity period of the unique identifierand determining that the at least a portion of the image data should beobscured only if a present date or time is within the validity period orauthenticating the unique identifier with respect to selected authoritydata. The authority data can include a cryptographic key and theauthenticating step can include validating the unique identifier usingthe cryptographic key and a digital signature of the unique identifier.

According to various aspects, there is provided a method of providingsurveillance image data, comprising: capturing image data can include animage of a target; querying a database to receive a preference settingcorresponding to the target; determining obscurant data of at least aportion of the image data corresponding to the target using the receivedpreference setting; and selectively modifying the image data accordingto the determined obscurant data to provide the surveillance image data.

Optionally, the obscurant data can be determined further using thereceived image data. The at least a portion can correspond to the imageof the target. The image data can correspond to a visibility region. Thequerying step can include querying the database based on the visibilityregion to determine whether a target is present in the visibilityregion, and performing the determining and modifying steps for thetarget determined to be present. The querying step can include providingto the database coordinates of a visibility polygon corresponding to thevisibility region. The querying step can include receiving from thedatabase data a masking layer representing one or more area(s) to bemasked and the determining step can include determining coordinates inthe image data corresponding to the area(s). The method can furtherinclude receiving an identity of the target and storing the receivedidentity in association with the preference setting in the database.Multiple sets of preference setting and target identity can be receivedand stored in the database. The method can further include receiving aunique identifier corresponding to the target, the querying step caninclude transmitting the unique identifier to the database. Thereceiving-identifier step can include transmitting a radio-frequency(RF) interrogation signal and receiving an RF identification signal inresponse, the RF identification signal can include data of the uniqueidentifier. The receiving-identifier step can include receiving theunique identifier from a location provider. The method can furtherinclude locating a tag of the target in the image data and decoding atarget identifier of the tag, the target identifier visually representedin the image data. The querying step can include transmitting thedecoded target identifier to the database. The determining step caninclude determining the obscurant data using the received image data andthe received preference setting. The method can further comprisereceiving a validity period of the unique identifier and determiningthat the at least a portion of the image data should be obscured only ifa present date or time is within the validity period or authenticatingthe unique identifier with respect to selected authority data. Theauthority data can include a cryptographic key and the authenticatingstep can include validating the unique identifier using thecryptographic key and a digital signature of the unique identifier.

According to various aspects, there is provided a surveillance devicecomprising a processor adapted to receive image data, identificationdata of a target visible in the image data, and a preference settingcorresponding to the identification data; and to selectively modify atleast a portion of the image data corresponding to the target accordingto the preference setting. Optional features described above can also beused in combination with the surveillance device, e.g., obscuring thetarget in the image data, providing the modified image data using adelivery system, receiving the preference setting via a networktransceiver, or the identification data including a unique identifier orlocation of a target. The identification data can include a uniqueidentifier associated with a cryptographic signature or a validityperiod.

According to various aspects, there is provided a surveillance system,an image-capture device adapted to produce image data of a scene;comprising: a database adapted to store preference setting(s); and asurveillance device as described above.

Optionally, the surveillance device in the surveillance system can beadapted to receive the image data from the image-capture device. Theprocessor of the surveillance device can be adapted to receive theidentification data and to transmit a query to the database, the querycan include the identification data, and the database can be adapted toreceive the query from the processor and transmit a corresponding storedpreference setting to the processor. The image-capture device can befurther adapted to produce the associated data indicating a visibilityregion of the image-capture device, the database can be further adaptedto store the respective preference settings of one or more target(s) andrespective location(s) of those targets and to respond to the query withthe preference setting and location of a target within the visibilityregion, and the processor can be further adapted to modify the at leasta portion of the image data corresponding to the visibility region andthe location only if the preference setting indicates such modificationshould be performed. The database can be further adapted to receive auser input and a unique identifier of the target and to store the userinput as the preference setting corresponding to the unique identifier.The database can be further adapted to receive the unique identifier andthe location of the target and to store the received location inassociation with the received unique identifier, so that the databaseresponds to the query by determining one or more unique identifier(s)having location(s) within the visibility region. The surveillance systemcan include a location provider adapted to periodically provide thelocation of the target to the database. The identification data caninclude a unique identifier associated with a cryptographic signature ora validity period and the database can be further adapted to storeinformation relating to the cryptographic signature or the validityperiod.

According to various aspects, there is provided a method of providingsurveillance image data, comprising: receiving image data can include animage of a target; receiving a preference setting corresponding to thetarget; and selectively modifying at least a portion of the image datacorresponding to the target according to the preference setting toprovide the surveillance image data.

Optionally, the method can further include querying the database todetermine whether a target is present in the visibility region, andperforming the modifying step when the target is present or with respectto the target determined to be present. The querying to determinewhether a target is present can include providing to the databasecoordinates of a visibility polygon corresponding to the visibilityregion.

The modifying step can include obscuring the at least a portion of theimage data in response to a preference setting requesting privacy. Themethod can further include receiving the preference setting and anidentity of the target and storing the received setting and identity inthe database. The method can further include capturing the image dataand querying a database to receive the preference setting. The methodcan further include receiving an identifier of the target and validationinformation of the identifier and determining whether a use of theidentifier satisfies the validation information.

Various aspects described herein advantageously provide systems for andways of determining that a person or other object should not be trackedby a drone or other optoelectronic surveillance or tracking device, andof blocking data captured by such drones or devices. Various aspectsprovide that a person or other object should be rendered more visible onsurveillance imagery. Various aspects advantageously permit individualsto select the level of privacy they desire with respect to systemsdescribed herein or systems implementing methods described herein.Various aspects operate using devices, e.g., cellular telephones, thatusers may already have. Various aspects use no personally-identifiableinformation, so that the identity of a person requesting privacy is notstored. Various aspects include storing preferences regarding privacy ina database.

Various aspects include storing preferences regarding privacy in amemory in a tag, or encoding those preferences in the structure, shape,or color of a tag or in a detectable (e.g., human- or machine-visible)pattern arranged over the surface of the tag.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other objects, features, and advantages of the presentinvention will become more apparent when taken in conjunction with thefollowing description and drawings wherein identical reference numeralshave been used, where possible, to designate identical features that arecommon to the figures, and wherein:

FIG. 1 shows an imaging platform and an imaging system according tovarious aspects;

FIG. 2 shows an exemplary video frame captured by a capture system;

FIGS. 3 and 4 show an exemplary video frame captured by a capture systemaccording to various aspects;

FIG. 5 shows an exemplary video frame and imaging platform according tovarious aspects;

FIG. 6 is an exemplary axonometric view of a residential neighborhood;

FIG. 7 is an exemplary masking layer corresponding to FIG. 6;

FIG. 8 is an example of the result of applying the masking layer of FIG.7 to the view of FIG. 6;

FIG. 9 is a high-level diagram showing the components of an exemplarydata-processing system;

FIGS. 10-12 show various examples of operation of imaging system 190,receiver 130, and related components;

FIG. 13 shows an exemplary video frame and imaging platform according tovarious aspects;

FIG. 14 is a dataflow diagram of exemplary systems for producingobscured image data;

FIG. 15 shows a flowchart illustrating exemplary methods of providingobscurant data; and

FIG. 16 shows a flowchart illustrating exemplary methods of providingsurveillance image data.

The attached drawings are for purposes of illustration and are notnecessarily to scale.

DETAILED DESCRIPTION

In the following description, some aspects will be described in termsthat would ordinarily be implemented as software programs. Those skilledin the art will readily recognize that the equivalent of such softwarecan also be constructed in hardware, firmware, or micro-code. Becauseimage manipulation algorithms and systems are well known, the presentdescription will be directed in particular to algorithms and systemsforming part of, or cooperating more directly with, systems and methodsdescribed herein. Other aspects of such algorithms and systems, andhardware or software for producing and otherwise processing the imagesignals involved therewith, not specifically shown or described herein,are selected from such systems, algorithms, components, and elementsknown in the art. Given the systems and methods as described herein,software not specifically shown, suggested, or described herein that isuseful for implementation of any aspect is conventional and within theordinary skill in such arts.

Various aspects use a cell phone as a transponder by registering thephone's location in a registry or database. This can be done using anapp that runs on the cell phone and updates the registry periodically,as discussed herein. Various aspects described herein advantageouslytest identified targets against the database. Targets that should beblocked are obscured by cones (or other shapes) of non-observation addedto the image data, or are otherwise blocked from view. In variousaspects, properties or settings, e.g., user preferences, are stored withthe location in the registry.

Various aspects use materials of a certain shape or colors of a certainpattern to anonymously identify targets to be blocked from view. Some ofthese aspects do not use a phone as a transponder and do not use aregistry. Areas of a meaningful pattern or color in the image data areadvantageously obscured, as are image-data features around them. Variousaspects keep both the unmodified image data and the modified image datathat result from obscuration or blocking of targets. Various aspectsprovide indicators that enhance the image of a target or other object toimprove visibility or to capture other properties of the target such asidentification or direction of travel.

FIG. 1 shows an imaging platform and an imaging system according tovarious aspects. Imaging platform 100 include, e.g., a drone such as anairplane UAV. Imaging platform 100 can also include any other type ofdrone or a fixed camera, e.g., a mall security camera, bridge-inspectioncamera, or traffic-light camera. Imaging platform 100 can also include aportable surveillance device, e.g., GOOGLE GLASS or another head-mountedimage-capture device, a video camera or other hand-held image capturedevice, or another electronic image-capture device, whether or notdesigned to operate in the visible light range of ˜400-700 nm. Invarious aspects, imaging platform 100 is autonomous, so processor 186controls the operational functions of imaging platform 100. In otheraspects, imaging platform 100 is controlled from a ground station.Imaging platform 100 can be a module connected to another system such asa passenger airplane, and likewise throughout. Imaging platform 100 canhave a total weight of, e.g., 2-5 lbs, or several grams. Imagingplatform 100 can have a maximum length of, e.g., 2-3″. Imaging platform100 can include an airplane, such as a drone, a passenger airplane(piston or jet), or a military aircraft (e.g., a fighter, bomber,reconnaissance airplane, or combination thereof; a microlight such as aninsect-sized drone; a blimp; a free balloon; or other configurations).Imaging platform 100 can include a helicopter, a robot, or a missile.Imaging platform 100 can include a ground vehicle, water craft orunderwater craft, e.g. an automobile, ship or submarine, respectively,or a space craft or a satellite. Imaging platform 100 can be remotelycontrolled (non-autonomous), autonomous, or semi-autonomous. Forexample, a semi-autonomous drone can navigate using on-board sensors andcomputers along a sequence of waypoints provided by a ground controller.In another example, if contact with the ground controller is lost, asemi-autonomous drone can fly a holding pattern at its present locationor a selected location, or can follow a predetermined path to a landingfield. In another example, the drone is a blimp with a propulsion unit,and if contact with the ground controller is lost, the drone uses thepropulsion unit to remain substantially stationary.

Imaging platform 100 includes image-capture device 110, e.g., anoptoelectronic device such as a CMOS or CCD image sensor, or aspectrophotometer or spectroradiometer. Image-capture device 110provides the captured image to processor 186. Image-capture device 110can include optical or digital zoom devices. Processor 186 can alsoreceive data from optional sensor 120, e.g., an RF sensor, as discussedbelow. Sensor 120 can include transmit electronics controlled byprocessor 186. In the example shown, image-capture device 110 and sensor120 are located in wing pods mounted on an aircraft drone. Image-capturedevice 110 or other sensors, e.g., sensor 120, can be mounted on thebottom or top of, or elsewhere on, or located in the fuselage, on thetail, or in other locations, of an aircraft drone or other imagingplatform 100.

Processor 186 transmits the received image from image-capture device 110via communications device 187. In the example shown, communicationsdevice 187 includes an antenna, but it can also include a wiredtransceiver, e.g., an Ethernet transceiver. The received image istransmitted via data link 188 to communications device 137. In thisexample, data link 188 is wireless, e.g., GSM, WiFi, free-space laser,or LNS public-safety radio in the 800 MHz or 900 MHz band, or anotherUHF or VHF band, and communications device 137 includes an antenna. Datalink 188 can also include a cable or optical fiber and communicationsdevice 137 a transceiver. Data link 188 can include a communication linkincluding a physical connector, such as an optoelectronic communicationwire, or including a non-physical connector such as a wirelessconnection, for example a radio or microwave link. Data link 188 canalso Data link 188 can be encrypted, or data transmitted over data link188 can be encrypted (e.g., using the Secure Sockets Layer, SSL,Internet protocol or other secured-transfer protocols), or both.

Receiver 130 receives the image data from communications device 137.Receiver 130 can include, e.g., a ground station for a drone or acontrol terminal for a security system. Receiver 130 can dispatch thereceived image data to storage 131 (e.g., a hard-disk drive or writableoptical drive), display 132 (e.g., an OLED or CRT display), or otherdevices (e.g., other processors, controllers, or drones). Dispatch canbe carried out over other network links. Receiver 130 can include apersonal computer or embedded system. Receiver 130 can be under thecontrol of an individual, e.g., person 222 (FIG. 5), or the operator ofimaging platform 100.

Image-capture device 110, optional sensor 120, processor 186,communications device 187, data link 188, communications device 137 andreceiver 130 together compose imaging system 190. Imaging system 190delivers captured images to desired outputs, e.g., storage 131 ordisplay 132. For example, receiver 130 can include a drone groundstation, and personal computer 136 can be connected over the Internet oranother network to receiver 130 to permit a person or computer system toaccess data from imaging system 190. Delivery system 139 can deliver theimage data, and can include a storage or display interface, networklink, Internet connection, or other device for connecting to a devicethat is to receive the delivered data. The imaging platform 100 caninclude components not part of imaging system 190, e.g., a propulsionsystem.

Imaging system 190 can include processing components. Processor 186 orreceiver 130, or other components of imaging system 190, can modifyimage data before it is delivered, or can produce additional outputsoverlaid on, displayed along side, or delivered with the image data. Theterm “processing component” refers to any processor, controller,microcontroller, firmware, hardware, or programmable logic in imagingsystem 190 capable of performing operations described herein. The term“processor” can include multiple such devices connected by a data linkand operating together to perform a function described herein.

FIG. 2 shows an exemplary video frame 200 captured by a capture system.Everything herein related to video frames also applies to still images,and vice versa. Walking on surface 210 (e.g., a sidewalk) are persons221,222,223,224. Receiver 130, or other components of imaging system190, processes image data for the image using analysis routines inhardware, firmware, software, or a combination. The analysis routineshave determined, in this example, that persons 222 and 223 are ofinterest, as discussed below. Overlaid on the video frame are indicators232,233. These can be shaped like ovals, rectangles, or other shapes,and can be steady, colored, glowing, blinking, noise (e.g., similar tothe “snow” or “static” visible on an NTSC color TV tuned to a deadchannel) or otherwise visually distinct from the image data. Persons 221and 224 are not of interest, so no indicators are shown for them. Theanalysis routines can determine the size of indicators 232,233 byinspection of the image data, e.g., to determine what portion(s) of theimage data is (are) moving and is (are) a different color than thebackground (surface 210).

In various aspects, processor 186, receiver 130, or another component ofimaging system 190 can analyze the captured image data to locate personsof interest or other objects or features of interest, and can outputdata that will result in indicators 232, 233 being stored on storage 131or displayed on display 132 (all FIG. 1). Software from ESRI or othervendors can be used to detect features of interest in a frame based oncolor. In video captures, adjacent frames can be compared and movingimage-data features can be evaluated for whether they are of interest.Comparing successive image frames can also be used to estimate the sizeof the object potentially of interest. The determined size of the objectof interest can be used to determine the size of indicators 232, 233.These aspects can be combined with aspects using tags 542,543, which arediscussed below.

FIG. 3 shows an exemplary video frame 300 captured by a capture systemaccording to various aspects. Surface 210 and persons 221,222,223,224are as shown in FIG. 2, as is indicator 233. However, person 222 isbearing tag 342. Tag 342 is visible in the image data captured byimaging system 190 and indicates to the capture system that image dataof the bearer of tag 342 should not be available for use. Obscurant 332is displayed over person 222. Obscurant 332 can be solid black, colored,glowing, blinking, or otherwise visually distinct from the image data.Obscurant 332 can be, e.g., an oval, rectangle, or other shape, e.g.,the same shape and sizes as indicator 232 (FIG. 2). Obscurant 332 canmodify the image data by superimposing content, as in the previoussentences, or by blurring, smearing, or otherwise distorting the imagedata of person 222 to render it unrecognizable. The effect of anyobscurant described in this disclosure is to modify, in some way, theimage data eventually displayed to a person watching the surveillancevideo, so that person 222 is not readily identifiable from the videoseen by the person watching. Persons 221 and 224 are not of interest, sono indicia are shown. Obscurant 332 can be applied by any processingcomponent of imaging system 190, e.g., processor 186 or receiver 130.Obscurant 332 can also be applied by display 132 in response to dataprovided by imaging system 190 indicating the size and position ofobscurant 332.

Obscurant 332 is shown hatched here so that tag 342 is visible.Obscurant 332 can also be opaque, showing none of the image dataobscured by it. As long as tag 342 is visible in the image data fromimage-capture device 110, a selected component of imaging system 190will apply obscurant 332. This can advantageously be performed withoutrequiring any database or information about the object or person ofinterest. Person 222 thus is shielded, e.g., from surveillance carriedout without a search warrant, by wearing or carrying tag 342. Person 222does not have to register with any database or agency, and does not haveto take any action other than bearing tag 342 to increase his privacy.In some aspects, video frame 300 including obscurant 332 is stored, sono data is stored regarding person 222. In other aspects, frame 300 isstored, and data corresponding to obscurant 332 are separately stored.This permits normal display of frame 300, in which person 222 is notvisible. This also permits display of stored frame 300 without obscurant332, e.g., when ordered by a court.

Tag 342 include material of a particular color or that emits or reflectsa particular wavelength of light. Tag 342 can also be marked with aparticular pattern, e.g., a ID or 2D barcode such as a QR code. Person222 can wear tag 342 as a badge on a lanyard around his neck, orattached to a belt loop. Tag 342 can be credit-card- or business-card-or key-card-sized. Person 222 can wear tag 342 as a pin or badge clippedor otherwise affixed to an article of clothing, e.g., a hat or a lapel.Tag 342 can be ˜1″×1″. Tag 342 can be an integral part of an article ofclothing. For example, just as hunters wear hunter-orange clothing,person 222 can wear a hat, shirt, or other article of clothing that is aspecific color, or that includes an area of a specific color. The largertag 342 is, the greater range at which the image of tag 342 captured byimage-capture device 110 will be at least one pixel in size, orotherwise detectable by processor 186 or another processing device.Imaging system 190 cannot respond to tag 342 if tag 342 is notdetectable in a given video frame 200, given the size of tag 342, theresolution of image-capture device 110, and the distance betweenimage-capture device 110 and tag 342. In an example, ahelicopter-mounted image-capture device 110 can have a resolution of atleast 1280×720 over an area ˜1 yd² at a distance of ˜15 mi. In anotherexample, an image-capture device 110 on a drone can have a visual rangeof ˜9 mi. when the drone is flying at ˜15,000′ (˜4.572 km) AGL. Invarious aspects, tag 342 carries other encoded information, e.g., thetail number of an aircraft or the car number of a police car. Imagingsystem 190 can recognize and decode this additional information, andtransmit it to receiver 130, database 599 (FIG. 5), or other components,along with image data or obscurant data, or separately from those typesof data. For example, the imaging system 190 can determine the unique IDvalues of mobile telephones in its field of view, communicate all thedetermined ID values to receiver 130, and only obscure or indicate theareas around those mobile telephones that correspond to targetsexpressing a preference (e.g., in database 599) with respect tosurveillance.

In various aspects, imaging system 190 responds to tag 342 as long asthe color of tag 342 covers at least one pixel, or covers enough ofseveral adjacent pixels to be discernable. In various aspects, tag 342blinks or strobes in a wavelength visible to image-capture device 110,which wavelength can be visible to humans or not.

FIG. 4 shows exemplary video frame 400. This is as video frame 300, butobscurant 332 is completely opaque. As a result, person 222, who isbearing tag 342, is not visible to a person watching video frame 400 ondisplay 132. In some aspects, storage 131 receives video frame 400including obscurant 332, so no data is stored regarding person 222. Inother aspects, storage 131 stores frame 200, and separately stores datacorresponding to obscurant 332. This permits normal display of frame400, in which person 222 is not visible. This also permits display ofstored frame 200 without obscurant 332, e.g., when ordered by a court.

In various aspects, different colors or patterns of tag 342 can havedifferent meanings. In an example, one color or pattern can mean “do nottrack me,” as shown in FIGS. 3-4. Another color can mean “please dotrack me.” Another color can be a distress signal. Tag 342 can have twosides or configurations. Person 222 can therefore adjust theconfiguration of tag 342 or flip tag 342 over to change fromdo-not-track to please-help-me mode. Imaging system 190 will indicatethat tag 342 in a distress or please-help color indicates a person ofinterest and will apply an indicator similar to indicator 233. In anexample, policemen or security guards can wear tags 342 of a particularcolor different from the do-not-track color. This permits readilylocating them in captured images. For example, if security guards at asporting event, e.g., a baseball game, wear tags 342 of a particularcolor or pattern, imaging system 190 will be able to readily locate themin captured images, even when facial recognition technology would beoverwhelmed by the number of faces in close proximity in the image.

Tags 342 can be placed on objects, plants, or animals in addition to onpeople. Tag 342 can be applied, e.g., to a house or car. In variousaspects, data from imaging system 190 is available to a person usingpersonal computer 136, e.g., using a database client such as a SQLclient, or a Web browser, e.g., FIREFOX. That person can place a “pleasetrack” tag on his house to cause imaging system 190 to record data aboutthe tagged house every time imaging platform 100 captures an image ofthe tagged house. This permits the person to determine the times andfrequency of drone visits.

FIG. 5 shows video frame 500 and imaging platform 100 according tovarious aspects. Imaging platform is as in FIG. 1, and includes sensor120. Video frame 500 is as frame 300 (FIG. 3) except as indicated anddescribed herein. Data links are shown dashed for clarity.

Person 222 is wearing, carrying, or otherwise bearing tag 542, e.g.,attached to or as part of an article of clothing, or in a pocket, purse,handbag, briefcase, suitcase, or other article of luggage. Tag 542 canbe visible to image-capture device 110, but that is not required. Invarious aspects, tag 542 is not visible to image-capture device 110. Tag542 can include a data terminal (“DT”), e.g., a cellular telephone, or abeacon.

Tag 542 communicates periodically with database 599, e.g., every second,10 seconds, 30 seconds, minute, two minutes, five minutes, or anotherinterval. The interval can be configured by a user, e.g., person 222.The user can configure the interval based on his velocity and desiredbattery life. If the user is moving quickly, e.g., on a motorcycle,using a shorter interval provides improved privacy since obscurant 332more closely tracks the location of person 222. However, more-frequentupdates can drain a battery of tag 542 more quickly than less-frequentupdates. Tag 542 can include a user interface, e.g., a touchscreen orscroll wheel, that permits person 222 to change the interval. Tag 542can also interact with a World Wide Web (WWW) server or smartphone appthat relays a command from the user to tag 542 to change the interval.In aspects in which tag 542 includes a smartphone, an interval-changingapp can run on tag 542.

In various aspects, tag 542 includes a unique identifier, e.g., a SIMcard number or MAC address. The unique identifier of tag 542 can becorrelated with the identity of person 222 (as would be, e.g., acell-phone number), or not (e.g., a universally-unique identifier, UUID,or globally unique identifier, GUID, uniquely created for the tag andnot correlated in any database with any information about person 222).Tag 542 also includes a location-sensing device, e.g., a GPS receiver orantenna (e.g., cell-phone-tower or WiFi-hotspot) triangulator. Invarious aspects, periodically, tag 542 determines its location andupdates database 599 with the determined location and the uniqueidentifier. In other aspects, a service provider periodically updatesdatabase 599 with the location of tag 542. In an example, tag 542includes a cellular telephone or other device communicating via acellular network. The provider of cellular service periodicallytriangulates the location of tag 542 using multiple base-stationantennas and updates database 599 with the triangulated location and theunique identifier (e.g., telephone number or SIM card number) of tag542. This can be automatically performed by a processor operated by theservice provider. Antennas or other detection units not part of imagingplatform I00 and useful for triangulating or otherwise determining thelocation of tag 542 are represented graphically as location unit 555.

Database 599 also includes a do-not-track flag for each uniqueidentifier. Database 599 can include other flags for each uniqueidentifier, and can be organized for rapid searches by location. “Flags”can be binary values (on, applied, or set vs. off, not applied, orclear), integers, strings, or other data types. For example, setting orapplying the do-not-track flag indicates a desire not to be tracked.Clearing the flag indicates a lack of that desire, but does notnecessarily indicate an affirmative desire to be tracked. Such a desirecan be indicated by a please-track flag. In various aspects, however,binary flags can be used to indicate affirmative desires. For example, a“tracking” flag can be set to indicate a desire to be tracked and clearto indicate a desire not to be tracked.

In various aspects, instead of tag 542, a remotely-detectable biometricor physical property serves to identify targets. For example, humans canbe identified through face recognition, pets can be recognized throughautomated recognition of fur patterns, and buildings can be identifiedthrough automated recognition of their shape and color scheme. Manybuildings, such as the Empire State Building in New York City, theSydney Opera House, the Esplanade Concert Hall (the “Big Durian”) inSingapore, have unique and recognizable shapes. Just as tags 342, 542can be recognized by shape or color, and information can be decoded fromthe captured image of a tag 342, 542, targets can be recognized by shapeor color, and information decoded from the captured images of thosetargets. In addition to or instead of a unique ID such as a phonenumber, data representing a person's face or other visually-discernable(to human or machine) characteristic of a target, e.g., shape or colorof the target or a tag on the target, can be used as a unique ID toquery the database for a preference setting corresponding to the target.

As imaging platform 100 operates, it communicates via a link (e.g., awireless link through communications device 187) with database 599.Imaging platform 100 periodically communicates to database 599 thelocation(s) currently in view of image-capture device 110. For example,processor 186 can query the database with a visibility region ofimage-capture device 110, i.e., the region the image-capture device 110can see. The visibility region can be communicated as a visibilitypolygon, each node of which is located at specific (e.g.) WGS84 latitudeand longitude values (and optionally elevation values). Locations insidethe visibility polygon are visible in the image captured byimage-capture device 110. The visibility polygon can be a quadrilateralor cone. The visibility polygon can include arcuate segments, eitherexpressed exactly or approximated with a plurality of nodes.

Database 599 responds to the visibility-polygon query from processor 186with the coordinates of each tag that is located in the visibilitypolygon and to which the do-not-track flag has been applied. In theexample shown, database 599 provides the coordinates of tag 542 toprocessor 186. Processor 186 then modifies the image data, or producesappropriate data, to provide obscurant 332. Obscurant 332 can be ageneric shape (and size), a shape or size based on the distance betweenimage-capture device 110 and tag 542, a shape or size stored in database599, or any combination thereof. Processor 186 can transmit theobscurant data or modified image data, e.g., via the communicationsdevice 187, or can store the obscurant data or modified image data forlater downloading or processing.

In various aspects, person 222 can use personal computer (PC) 586 toapply or remove the do-not-track flag or other flags or information indatabase 599. Database 599 can thus be a registry of privacyinformation.

In other aspects, imaging platform 100 can determine locations andunique IDs of targets in its field of view, e.g. using sensor 120.Imaging platform 100 can then query database 599 with those IDs, insteadof or in addition to querying the database with a visibility polygon.The result of a query can be a preference setting, and imaging platform100 can provide obscurant data or modified image data corresponding tothe received preference settings and determined locations. In otherexamples, imaging platform 100 transmits the unique IDs and locations,or the preference settings and locations, to receiver 130 or othercomponents of imaging system 190; the component receiving theinformation can provide the obscurant data or modified image data.

In various aspects, database 599 is not used. Instead, imaging platform100 interrogates the areas in the visibility polygon using sensor 120,which can include a transceiver. Tag 542 detects an interrogation signalfrom sensor 120 and responds with its coordinates and do-not-track flagsetting. In various examples, tag 542 only responds if the do-not-trackflag has been applied; in other examples, tag 542 responds whether ornot the flag has been applied. Processor 186 receives tag 542's datafrom sensor 120 and applies obscurant 332 (or produces correspondingdata) corresponding to the reported location from tag 542.

In various aspects, tag 542 includes a transponder. Tag 542 responds tothe interrogation signal from sensor 120 with its unique ID. Processor186 queries database 599 with the received unique ID. If thedo-not-track flag has been applied to that unique ID, obscurant 332 isapplied (or corresponding data are produced). Database 599 can beupdated to indicate that tag 542 was observed at a particular place andtime by a particular imaging platform 100. Each imaging platform 100 canhave a unique ID (e.g., for aircraft, tail number; or, in general, GUIDor other unique ID). In various aspects, sensor 120 includes a radarthat can query transponders in range.

In various aspects, tag 542 includes an RFID tag and sensor 120 includesan RFID reader adapted to read tag 542. In various aspects, thetransponder in tag 542 broadcasts the location of tag 542. In variousaspects, sensor 120 estimates the location of tag 542, as describedherein. In various aspects, multiple imaging platforms 100 can combineresults from sensor 120 to estimate locations of tags 542. For example,three imaging platforms 100, each with a GPS or other device thatprovides the location of imaging platform 100 to respective processor186, can communicate with each other to triangulate the location of tag542 within the detection range of each of them.

In other aspects, tag 542 broadcasts its location, do-not-track status,and optionally its unique ID periodically, intermittently, orcontinually. Processor 186 listens for those broadcasts using sensor 120and applies an obscurant or provides data relating to an obscurant.Processor 186 can, additionally or alternatively, update an internaldata store with the received locations of tags 542 in the visibilitypolygon, and produce obscurants or obscurant data using the informationin the data store. The contents of the data store can be transmittedthrough imaging system 190. Any processing component of imaging system190, e.g., processor 186, can interpolate motion between successivebroadcasts from a given tag 542 to estimate where the obscurant shouldbe located at any given time. Alternatively, the obscurants can remainfixed at the last-received position.

Aspects in which tags 542 broadcast, or respond to sensor 120 with, onlytheir location and flags, and not their unique identifiers,advantageously reduce the need for storage of personally-identifiableinformation about the movements of person 222. Some aspects usingdatabase 599 can advantageously permit person 222 to adjust droneactivity as desired by updating the database. This can permit using asimple, robust tag 542 with no user interface.

In various examples, sensor 120 snoops signals from mobile telephones orother portable electronic devices near the imaging platform 100, orinterrogates such devices, e.g., using a base station included inimaging platform 100. Sensor 120 determines the location of each devicelocated, e.g., by rasterizing a directional antenna across the field ofview of the image-capture device 110. The imaging platform 100 (or,e.g., the receiver 130) then retrieves the preference setting for thedetermined unique ID from the database. In these examples, the database599 can store target locations or not, and in some examples, database599 holds no location data. Processor 186 or, e.g., receiver 130, thenproduces obscurant data or modifies image data.

In various examples, such devices, e.g., tag 542, update locations indatabase 599. Imaging platform 100, e.g., a drone, queries database 599with a visibility region or other field-of-view data, and database 599responds with the locations to be obscured or indicated. This permitsbuilding a drone with simpler, possibly less costly electronics. Inthese and other examples throughout, database 599 can include locationsof untagged targets to be obscured or indicated. Targets do not have tobe tagged as long as they are in the database. Such untagged targets caninclude buildings or other fixed structures or areas, or can includemoving targets that periodically update database 599, e.g., using asmartphone app as described below.

In various examples, database 599 can indicate that person 222 bearingtag 542 wishes to know the last time a drone observed tag 542.Information regarding the last-observation time can be stored indatabase 599 by processor 186 while imaging platform 100 is operating,and can be retrieved by person 222 using PC 586. The last-observationtime can be stored by a command from the processor or autonomously bydatabase 599, e.g., using a trigger on queries for flags of tag 542.Tags 542 can also be applied to non-humans, as discussed above. In someaspects, database 599 can thus be queried for the last time imagingplatform 100 observed a house bearing a record-visit-timestamp tag.Flags in database 599 can also cause processor 186 to update database599 with information about the location at which tag 542 was mostrecently observed, or with a photo of the bearer of tag 542 at the timeof observation (a crop from the captured image). The timestamp can havea desired granularity, e.g., year, month, day, minute, second, ormillisecond. The timestamp can correspond, e.g., to the image datacaptured by the drone or other imaging platform 100 observing tag 542.

In any of the aspects described herein, data can be stored in storage131 or database 599 for a selectable amount of time, e.g., 180 days.Database 599 can include flags directing data to be stored longer. Whenretrieving captured images from database 599 or storage 131, the flagsin database 599 can be consulted to narrow down the search rapidly tounique IDs with do-track fields set. In an example, parents can set thedo-track fields on tags carried by their children so that they will havea record of their children's movements.

As discussed above, in some aspects, storage 131 stores frame 200, andseparately stores data corresponding to obscurant 332. Normally, imagedata are only provided with obscurant 332 in place. In various aspects,storage 131 stores the unique ID associated with a given obscurant 332.An interface (not shown) receives a user input indicating that the userwishes to voluntarily waive blocking of the unique ID of his tag at aspecific place or time, or in a given range of places or times, or ingeneral. Storage 131 then provides image data without obscurant 332 inthe desired place or time or range thereof. Imaging system 190 can alsoinclude these functions. Receiver 130 can receive the user input andprovide to display 132 image data in which obscurant 332 is in placeexcept in the selected place(s) or time(s).

In various aspects, video frame 500 includes do-track features ofinterest, or both do-track and do-not-track features of interest. In theexample shown, person 223 is wearing or carrying do-track tag 543, andso is highlighted with indicator 233. If obscurant 332 and indicator 233overlap, obscurant 332 can override the overlapping portions ofindicator 233, indicator 233 can override the overlapping portions ofobscurant 332, the overlap area can be bisected to reduce the size ofboth obscurant 332 and indicator 233, or a choice of which action totake can be given to an operator of imaging system 190. In variousaspects, whether person 222 or person 223 is closer to image-capturedevice 110 can be used to make the determination. For example, if person222 is closer, image data for person 222 will obscure person 223naturally. In this situation, the choice can be made that indicator 233can override the overlapping portions of obscurant 332. Some of person222 will be visible as a result, but much of person 222 will be blockedfrom view by person 223's body. This provides person 222 with moreprivacy than he would have had without tag 542, and still preserves thetracking that person 223 wants.

In various aspects, tag 542 either broadcasts or responds with apresence indication or one or more flags rather than a location. In anexample, tag 542 is an RFID tag without location-receiving circuitry.Sensor 120 can include a directional antenna or a time-of-flightmeasurement unit to determine slant range to tag 542. Sensor 120 caninclude multiple spatially-separated antennas to permit triangulatingthe location of tag 542, or improving the accuracy of an estimate of tag542's position. Sensor 120 or processor 186 can query triangulationground stations, e.g., cell-phone base stations, to determine thelocation of tag 542. When sensor 120 detects or receives a response fromsuch a tag 542, processor 186 deduces that there is an object in thevisible polygon. If tag 542 only supplies a presence indication,processor 186 can assume a default state of the flags, e.g.,do-not-track applied. Tag 542 can also return one or more flag(s)configurable by person 222, e.g., using DIP switches on tag 542.Processor 186 infers a size and shape for obscurant 332 (or anindicator) based on information received from sensor 120 about thelocation of tag 542. In an example, obscurant 332 covers the entirety ofany frame of captured image data containing an RFID do-not-track tag542. In another example, sensor 120 scans or otherwise determines theazimuth and elevation angles of tag 542 with respect to sensor 120, andprocessor 186 obscures or produces obscurant data for only a portion ofthe image lying in that direction from sensor 120. Processor 186 canadjust the angles, if necessary, to compensate for different locationsof image-capture device 110 and sensor 120.

FIGS. 6-8 show the operation of other aspects. FIG. 6 is an axonometricview of a residential neighborhood. The neighborhood shown has houses611, 612, 613, and 614 on either side of street 620. This view isrepresentative of an oblique perspective image of the neighborhoodcaptured by an exemplary drone.

The grid lines in FIG. 6 represent coordinates. Perspective images canbeen geo-referenced or geo-rectified so that the latitude and longitudeof each point in the view can be readily determined. Georeferencingincludes angle correction so that coordinates can be determined foroverhead images or oblique images. The grid lines can be parallel to thelines of latitude and longitude in a datum covering the neighborhood.The grid lines can be present in a captured image or added to it, ornot. Data representing the grid can be stored separately. House 612 islocated in grid square 5490.

FIG. 7 shows a masking layer on the same grid as FIG. 6. This isrepresentative of a layer used in a GIS (geographical informationsystem). The image displayed to the user of a GIS is a superimpositionof various layers, starting from a base map. According to variousaspects, the georeferenced masking layer represented in FIG. 7 issuperimposed over the image represented in FIG. 6.

FIG. 8 shows the result of that superimposition. The view is as in FIG.6, except that data for grid square 5490, including house 612, have beenobscured by obscurant 632. The shape and size of the obscurant shown isan example. The obscurant can be the shape and size of grid square 5490or house 613, or can be another size or shape determined as describedabove.

The masking layer of FIG. 7 can be stored in a database, e.g., database599. While imaging platform 100 is operating, it can query database 599with the visible polygon to retrieve appropriate masking layers.Processor 186 can produce obscurants or obscurant data using theretrieved masking layers. Other processing components of imaging system190 or storage 131 can also produce the obscurants or obscurant data.The masking layers in database 599 can be updated periodically, e.g.,monthly or quarterly. In various examples, the masking layer includesdata masking specific parcels. That is, instead of or in addition togrid squares, the masking layer includes shapes corresponding to theshapes and positions of specific parcels or pieces of property, orspecific fixed-position objects or areas. In various aspects, themasking layer can also include data specifying other flags, or otherlayers can be used. For example, a separate layer could includeplease-track flags analogous to those described above for tags 542.

In various examples, combinations of aspects described above are used.For example, a person's house can be marked “please track.” This permitsthe person to access data about when the house was imaged by an imagingplatform 100. The please-track marker can be applied by a please-tracktag, as discussed above with reference to FIG. 5, or by a please-tracklayer, as discussed with reference to FIG. 7. The person, however, canwear a do-not-track tag 342, 542 (e.g., a beacon), as described withreference to FIG. 3 or 5. As a result, a drone capturing an image of thehouse will obscure the person if he is walking around his backyard.Other combinations of do-track and do-not-track can be used.

In various examples, a user registers the unique ID of tag 542, or thecoordinates of a parcel, with database 599, and indicates thedo-not-track flag should be set. Processor 186, during operation ofimaging platform 100, queries database 599 to locate do-not-trackregions in the visible polygon. Processor 186 produces obscurants (e.g.,obscurant 332) or data indicating where obscurants should go. In theformer situation, the modified image is transmitted to receiver 130. Inthe latter situation, the unmodified image and the obscurant data aretransmitted to receiver 130.

Obscurants and indicators can be added by any processing component(s) ofimaging system 190, e.g., processor 186 on imaging platform 100, or by adownstream component such as storage 131. The term “targets” refers topeople, houses, other buildings, doors or windows of structures, plants,animals, objects, or any other matter or volume that can be associatedwith a flag using any of the ways described herein. Targets can beidentified, e.g., by bearing tags 342 or 542, or by having theirlocations stored in database 599.

FIG. 9 is a high-level diagram showing the components of adata-processing system for analyzing image data and performing otheranalyses described herein. The system includes a data processing system910, a peripheral system 920, a user interface system 930, and a datastorage system 940. The peripheral system 920, the user interface system930 and the data storage system 940 are communicatively connected to thedata processing system 910. Processor 186 and receiver 130 can eachinclude one or more of systems 910, 920, 930, 940.

The data processing system 910 includes one or more data processingdevices that implement the processes of the various aspects, includingexemplary processes described herein such as those shown in FIGS. 15 and16. The phrases “data processing device” or “data processor” areintended to include any data processing device, such as a centralprocessing unit (“CPU”), a desktop computer, a laptop computer, amainframe computer, a personal digital assistant, a Blackberry™, adigital camera, cellular phone, or any other device for processing data,managing data, or handling data, whether implemented with electrical,magnetic, optical, biological components, or otherwise.

The data storage system 940 includes one or more processor-accessiblememories configured to store information, including the informationneeded to execute the processes of the various aspects, including theexample processes described herein. The data storage system 940 can be adistributed processor-accessible memory system including multipleprocessor-accessible memories communicatively connected to the dataprocessing system 910 via a plurality of computers or devices. On theother hand, the data storage system 940 need not be a distributedprocessor-accessible memory system and, consequently, can include one ormore processor-accessible memories located within a single dataprocessor or device.

The phrase “processor-accessible memory” is intended to include anyprocessor-accessible data storage device, whether volatile ornonvolatile, electronic, magnetic, optical, or otherwise, including butnot limited to, registers, floppy disks, hard disks, Compact Discs,DVDs, flash memories, ROMs, and RAMs.

The phrase “communicatively connected” is intended to include any typeof connection, whether wired or wireless, between devices, dataprocessors, or programs in which data can be communicated. The phrase“communicatively connected” is intended to include a connection betweendevices or programs within a single data processor, a connection betweendevices or programs located in different data processors, and aconnection between devices not located in data processors. In thisregard, although the data storage system 940 is shown separately fromthe data processing system 910, one skilled in the art will appreciatethat the data storage system 940 can be stored completely or partiallywithin the data processing system 910. Further in this regard, althoughthe peripheral system 920 and the user interface system 930 are shownseparately from the data processing system 910, one skilled in the artwill appreciate that one or both of such systems can be storedcompletely or partially within the data processing system 910.

The peripheral system 920 can include one or more devices configured toprovide digital content records to the data processing system 910. Forexample, the peripheral system 920 can include digital still cameras,digital video cameras, cellular phones, or other data processors. Thedata processing system 910, upon receipt of digital content records froma device in the peripheral system 920, can store such digital contentrecords in the data storage system 940.

The user interface system 930 can include a mouse, a keyboard, anothercomputer, or any device or combination of devices from which data isinput to the data processing system 910. In this regard, although theperipheral system 920 is shown separately from the user interface system930, the peripheral system 920 can be included as part of the userinterface system 930.

The user interface system 930 also can include a display device, aprocessor-accessible memory, or any device or combination of devices towhich data is output by the data processing system 910. In this regard,if the user interface system 930 includes a processor-accessible memory,such memory can be part of the data storage system 940 even though theuser interface system 930 and the data storage system 940 are shownseparately in FIG. 9.

In view of the foregoing, aspects of the invention provide improvedcontrol by a person of data captured by surveillance systems thatrelates to that person. A technical effect is to provide a surveillancevideo stream that advantageously obscures people who do not which to becaptured on video. Another technical effect of various aspects is toprovide a surveillance video stream in which, because portions areobscured, un-obscured portions are more visible to the human eye, orstand out more prominently to a human observer.

As will be appreciated by one skilled in the art, aspects of the presentinvention may be embodied as a system, method, or computer programproduct. Accordingly, aspects of the present invention may take the formof an entirely hardware aspect, an entirely software aspect (includingfirmware, resident software, micro-code, etc.), or an aspect combiningsoftware and hardware aspects that may all generally be referred toherein as a “service,” “circuit,” “circuitry,” “module,” or “system.”Furthermore, aspects of the present invention may take the form of acomputer program product embodied in one or more computer readablemedium(s) having computer readable program code embodied thereon.

A computer program product can include one or more storage media, forexample; magnetic storage media such as magnetic disk (such as a floppydisk) or magnetic tape; optical storage media such as optical disk,optical tape, or machine readable bar code; solid-state electronicstorage devices such as random access memory (RAM), or read-only memory(ROM); or any other physical device or media employed to store acomputer program having instructions for controlling one or morecomputers to practice method(s) according to various aspect(s).

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific examples (a non-exhaustive list) of the computer readablestorage medium would include the following: an electrical connectionhaving one or more wires, a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an optical fiber,a portable compact disc read-only memory (CD-ROM), an optical storagedevice, a magnetic storage device, or any suitable combination of theforegoing. In the context of this document, a computer readable storagemedium may be any tangible medium that can contain, or store a programfor use by or in connection with an instruction execution system,apparatus, or device.

Program code or executable instructions embodied on a computer readablemedium may be transmitted using any appropriate medium, including butnot limited to wireless, wireline, optical fiber cable, RF, or anysuitable combination of appropriate media.

Computer program code for carrying out operations for aspects of thepresent invention may be written in any combination of one or moreprogramming languages. The program code may execute entirely on theuser's computer (device), partly on the user's computer, as astand-alone software package, partly on the user's computer and partlyon a remote computer or entirely on the remote computer or server. Inthe latter scenario, the remote computer may be connected to the user'scomputer through any type of network, including a local area network(LAN) or a wide area network (WAN), or the connection may be made to anexternal computer (for example, through the Internet using an InternetService Provider).

Computer program instructions can be stored in a computer readablemedium that can direct a computer, other programmable data processingapparatus, or other devices to function in a particular manner. Thecomputer program instructions may also be loaded onto a computer, otherprogrammable data processing apparatus, or other devices to cause aseries of operational steps to be performed on the computer, otherprogrammable apparatus or other devices to produce a computerimplemented process such that the instructions which execute on thecomputer or other programmable apparatus provide processes forimplementing the functions/acts specified herein.

(FIGS. 10-12 show various examples of operation of imaging system 190,receiver 130, and related components. Imaging system 190 is as shown inFIG. 1. FIG. 10 shows image-capture device 110 and optional sensor 120providing data to processor 186. Processor 186 transmits image data andoptionally associated data over data link 188 to receiver 130. Receiver130 provides data to storage 131 or display 132. Associated data caninclude data about targets or obscurants, e.g., shapes, layers, regions,masks, or polygons defining areas of the image that contain targets orthat should be obscured. In various aspects, data to and from storagepass through security unit 1056. Security unit 1056 can provide accesscontrol for storage 131, e.g., to restrict access to storage 131 toauthorized users. Security unit 1056 can also provide data-integritychecks or non-repudiation. For example, data in storage 131 can bestored with checksums, secure hashes, or cryptographic signatures (e.g.,RSA, PGP, or other public-key signatures, or private-key signatures)that permit detecting alterations to the stored data. Security unit 1056can keep timestamps or audit logs of accesses to storage 131. In variousaspects, security unit 1056 and storage 131 together providetamper-evident storage of data. This permits humans using the data toavoid relying on data that have been tampered with, e.g., as evidencedby a checksum failure. Security unit 1056 can also be used in theexamples shown in FIGS. 11 and 12, and in other examples herein.

FIG. 11 shows an example of obscuration being performed by processor 186on imaging platform 100. Processor 186 receives image data fromimage-capture device, and receives data about the locations of targetsfrom sensor 120, database 599, or both. Processor 186 appliesobscurant(s), e.g., obscurant 332, and supplies obscured image data toreceiver 130. Throughout FIGS. 11-12, empty circles represent operationsperformed by the component in which the circle is represented, theinputs and outputs of the operations represented by the arrows connectedto the empty circles.

FIG. 12 shows an example of processor 186 determining the obscurationand receiver 130 applying the obscuration. Processor 186 provides theimage data from image-capture device 110 to receiver 130 withoutobscuring it. (Processor 186 can modify the image data in other ways,e.g., georeferencing or adjusting resolution, contrast, or color.)Processor 186 produces obscurant data using the inputs from sensor 120or database 599, or both, and transmits the obscurant data to receiver130. Image data and obscurant data can be transmitted over separate datalinks or multiplexed on one data link. Receiver 130 supplies image dataand obscurant data separately to storage 131. Receiver 130 combines theimage data and the obscurant data to apply the obscurants to the image.The obscured image data is then supplied to display 132 using deliverysystem 139 (discussed above with reference to FIG. 1). In this way,without a court order or other human decision, only obscured data arevisible. With such an order or decision, un-obscured image data can beretrieved from storage 131. In various aspects, storage 131 does notstore obscurant data.

FIG. 13 shows an exemplary video frame 1300 and imaging platform 100according to various aspects. Imaging platform 100 includes, e.g., adrone such as an airplane UAV. Imaging platform 100 can also include anyother type of drone or a fixed camera, e.g., a mall security camera,bridge-inspection camera, or traffic-light camera. Imaging platform 100can be remotely controlled (non-autonomous), autonomous, orsemi-autonomous, as described above.

Imaging platform 100 includes image-capture device 110, e.g., anoptoelectronic device such as a CMOS or CCD image sensor, or aspectrophotometer or spectroradiometer. Image-capture device 110provides the captured image to processor 186. Image-capture device 110can include optical or digital zoom devices. Image-capture device 110 orother sensors can be mounted on the bottom or top of, or elsewhere on,the fuselage of an aircraft-drone imaging platform 100.

Processor 186 transmits the received image from image-capture device 110via communications device 187, which can be as described above. Receiver130 receives the image data from communications device 137, as discussedabove.

Image-capture device 110, processor 186, communications device 187, datalink 188, communications device 137 and receiver 130 together compose animaging system as described above. In various aspects, imaging platform100 is a “dumb drone,” i.e., a system that does not apply imageprocessing to provide obscurants or indicators as described below. Invarious aspects, data link 188 includes two parallel data streams: rawimage data in one stream, and the location of imaging platform 100 or ofvisible features in the other stream.

In various aspects, receiver 130 receives image data from imagingplatform 100 via data link 188 and produces video frame 500. Receiver130 can also communicate with database 599, as discussed below. Invarious examples, data link 188 is one-way from the imaging platform 100to the receiver 130. This advantageously reduces the complexity of theimaging platform 100, permitting these examples to be readily used with,e.g., microlight drones or passive surveillance cameras such as closed-circuit television (CCTV) cameras.

Video frame 1300 is an exemplary video frame produced by the imagingsystem according to various aspects. Surface 210, persons221,222,223,224, and other features are as discussed above withreference to video frame 500 (FIG. 5). In this example, persons 222 and223 are of interest. Overlaid on the video frame are obscurant 332 andindicator 233, as described above with reference to FIG. 5. Obscurant332 can be as described above with reference to FIG. 3 can modify theimage data by Persons 221 and 224 are not of interest, so no indicatorsor obscurants, or other indicia, are shown for them. The analysisroutines can determine the respective sizes of obscurant 332 orindicator 233 by inspection of the image data as described above.

Receiver 130 or another component of the imaging system can analyze thecaptured image data and provide outputs as described above withreference to FIG. 5. Analysis of image data can be used together withanalysis or processing based on tags 542, 543.

Still referring to FIG. 13, obscurant 332 is represented graphicallywith a hatching pattern to show the relationship between person 222, tag542, and obscurant 332. In various examples, obscurant 332 can behatched or can be opaque, showing none of the image data obscured by it.

Tags 542, 543 can be placed on objects, plants, or animals in additionto on people. Tag 542, 543 can be applied, e.g., to a house or car. Invarious aspects, different tags 542, 543 or configurations of tags canhave different meanings, as discussed below. In an example, one tag canmean “do not track me,” e.g., tag 542. Another tag can mean “please dotrack me,” e.g., tag 543. Another tag can be a distress signal. Tag 542can be visible to image-capture device 110, but that is not required. Invarious aspects, tag 542 is not visible to image-capture device 110. Tag542 includes a data terminal (“DT”), e.g., a cellular telephone or otheractive beacon.

In various aspects, tag 542 includes a location system adapted todetermine the location of tag 542 in a selected frame of reference. Thelocation system can include a GPS or GLONASS receiver, a triangulatorthat consults a database of known locations of nearby radio sources(e.g., cell-phone towers or WIFI hotspots). The location system can alsoinclude a transceiver that communicates with, e.g., a radar to receivelocation information determined by the radar. In an example, the radarpings tag 542 to determine azimuth, slant range, and optionallyelevation, then converts those to a location in the selected frame ofreference and sends them back to tag 542. In an example, tag 542includes a cellular telephone having hardware to determine its locationand software to report that location to a registry, e.g., database 599.

In other aspects, the location of tag 542 is determined outside tag 542.For example, tag 542 can include a cellular telephone or other devicecommunicating via a cellular network. The provider of cellular serviceperiodically triangulates the location of tag 542 using multiplebase-station antennas. This can be automatically performed by aprocessor operated by the service provider. Equipment not part of tag542 for detecting the location of tag 542 is represented graphically asoptional location unit 555. Unit 555 can include antennas or otherdetection units not part of imaging platform 100 and useful fortriangulating or otherwise determining the location of tag 542. Invarious aspects, unit 555 leverages the built-in Enhanced 911 (E911)location-determination unit that can determine the location of tag 542.Using E911 and related location technology (e.g., triangulation, globalpositioning system, GPS, assisted GPS, aGPS, or location fingerprintingusing known locations of WIFI routers or other RF signal sources), cell-phone carriers or other communications providers can provides locationinformation for tag 542 without requiring user intervention or action,e.g., without running a specialized smartphone app. In various aspects,unit 555 (e.g., a cellular telephone base station) receives a preferenceinput from the user (e.g., from database 599 or another database). Theuser can set the preference input to determine whether the location oftag 542 should be reported or not.

In this disclosure, the system that determines the location of tag 542is referred to as the “location provider.” The location provider can bepart of tag 542 or not, as discussed above.

The location provider communicates periodically or a periodically withdatabase 599, e.g., at intervals described above. The interval can beconfigured by a user, e.g., person 222. The user can configure theinterval based on his velocity and desired battery life. Tag 542 or thelocation provider can include a user interface, e.g., a touchscreen orscroll wheel, that permits person 222 to change the interval. Tag 542 orthe location provider can also interact with a World Wide Web (WWW)server or smartphone app that relays a command from the user to tag 542to change the interval.

In various aspects, tag 542 includes a unique identifier, as describedabove. The location provider determines the location of tag 542 andupdates database 599 with the determined location and the uniqueidentifier.

In various of these aspects, database 599 stores unique IDs andlocations received from location providers. Database 599 also stores ado-not-track flag for each unique identifier. Database 599 can storeother flags for each unique identifier, and can be organized for rapidsearches by location (e.g., using quadtree subdivision on latitude andlongitude coordinates). Flags can be as described above.

As imaging platform 100 operates, it communicates via a link (e.g., awireless link through communications device 187) with receiver 130.Imaging platform 100 periodically communicates to receiver 130 itslocation, or location(s) currently in view of image-capture device 110.For example, processor 186 can provide to receiver 130 a visibilityregion of image-capture device 110, i.e., the region the image-capturedevice 110 can see. The visibility region can be communicated as avisibility polygon, as described above.

Receiver 130 then queries database 599 with the visibility polygon.Database 599 responds to the visibility-polygon query from receiver 130with the coordinates of each tag that is located in the visibilitypolygon and to which the do-not- track flag has been applied. In theexample shown, database 599 provides the coordinates of tag 542 toreceiver 130. Receiver 130 then modifies the image data, or producesappropriate data, to provide obscurant 332. Obscurant 332 can be ageneric shape (and size), a shape or size based on the distance betweenimage-capture device 110 and tag 542, a shape or size stored in database599, or any combination thereof. In various aspects, processor 186queries database 599 with the visibility polygon. Processor 186 thenproduces the obscurant data, or modifies the image data, to provideobscurant 332. In various aspects, receiver 130 queries database 599with the unique ID of a tag 542 detected, e.g., using the sensor 120(FIG. 5).

In any of the aspects described herein, data can be stored in database599 for a selectable amount of time, e.g., 180 days. Database 599 caninclude flags directing data to be stored longer. When retrievingcaptured images from database 599 or storage 131, the flags in database599 can be consulted to narrow down the search rapidly to unique IDswith do-track fields set. In an example, parents can set the do-trackfields on tags carried by their children so that they will have a recordof their children's movements.

In various aspects, video frame 500 includes do-track features ofinterest, or both do-track and do-not-track features of interest. In theexample shown, person 223 is wearing or carrying do-track tag 543, andso is highlighted with indicator 233. Highlighting is performed byreceiver 130. In various aspects, when receiver 130 queries database599, database 599 responds with the locations of all tags in thevisibility polygon, and the corresponding flags. Receiver 130 inspectsthe flags and produces either obscurants or indicators at the correctlocations in video frame 500. Imaging platform 100 or receiver 130 cangeoreference or otherwise correlate pixel locations in an image fromimage-capture device 110 with the visibility polygon to determine wheregiven coordinates appear in video frame 500.

In various examples, a user registers the unique ID of tag 542, or thecoordinates of a parcel, with database 599, and indicates thedo-not-track flag should be set. Receiver 130, during operation ofimaging platform 100, queries database 599 to locate do-not-trackregions in the visible polygon. Receiver 130 produces obscurants (e.g.,obscurant 332) or data indicating where obscurants should go.

Obscurants and indicators can be added by any processing component(s) ofthe imaging system, e.g., processor 186 on imaging platform 100 orreceiver 130, or by a downstream component such as a storage unit.

FIG. 14 is a dataflow diagram of systems for producing obscured imagedata according to various aspects. Image-capture device 110 providesimage data to processor 186. Optional capture-location provider 255provides a location to which the image corresponds, e.g., the GPScoordinates and orientation (roll, pitch, and yaw) of image-capturedevice 110 at the time the image is captured (orientation can bedetermined using accelerometers). In various aspects, processor 186 doesnot produce obscurants, but can modify the image data in other ways,e.g., georeferencing or adjusting resolution, contrast, or color.Processor 186 transmits image data 1488 and optionally associated data1489, e.g., the capture location from provider 255, over data link 188to receiver 130. Empty circles represent operations performed by thecomponent in which the circle is represented, the inputs and outputs ofthe operations represented by the arrows connected to the empty circles.

In various aspects, receiver 130 provides to storage 131 the image dataand, separately or separably, the associated data. Receiver 130 alsodetermines and applies obscuration(s) or indicator(s). Receiver 130queries database 599 with the associated data, or with visible-polygoninformation extracted from the received image data (e.g., locations oflandmarks in the image). As discussed above, database 599 can also bequeried with the unique ID of a tag. The associated data can thusinclude identification data of the tag 542. Receiver 130 processes theimage data according to the query results 1459 to provide obscured imagedata 1490. The term “obscured image data” includes images to which onlyindicators have been applied. The obscured image data is supplied todisplay 132, and can be stored in storage 131. In this way, without acourt order or other human decision, only obscured data are visible.With such an order or decision, un-obscured image data can be retrievedfrom storage 131. In various aspects, storage 131 does not storeobscurant data. As described above, e.g., with reference to FIG. 2,indicators can be added to the obscured image data instead of or inaddition to obscurants.

In various aspects, storage 131 includes one or more data-storagedevice(s), e.g., hard drives, that store both un-obscured image data andobscurant data. In various aspects, storage 131 includes data-storagedevice(s) that store un-obscured image data and separate data-storagedevice(s) that store obscurant data. Storage 131 can also store obscuredimage data, on data-storage device(s) that also hold un-obscured imagedata or obscurant data, or on separate data-storage device(s).

In various aspects, cryptographic or other protection is applied so thatnon-obscured image data is stored in storage 131 in a tamper-evidentmanner. Receiver 130 delivers obscured video or images to display 132,but raw data are available from storage 131 on court order or otherappropriate conditions.

In various aspects, imaging platform 100 captures images of parking lotsor other areas with a limited number of assignable areas (e.g., parkingspaces in a lot, camping sites in a park or campground, or hookups in anRV park). Processor 186, receiver 130, or one or more other component(s)of the imaging system analyze the image data to determine the locationsof available assignable areas. These locations, or directions to themfrom a user's location (e.g., the location of tag 542 that includes asmartphone), are presented to the user via tag 542 or another interface.In an example, a smartphone app presents directions to the nearestavailable assignable area. In various aspects, the smartphone apppresents image data with indicators, e.g., as shown in FIG. 13, when thenearest available assignable area is in view. In various examples,receiver 130 provides directions and images with indicators tosmartphone 585 or tag 542. In various examples, phone 585 is part ofreceiver 130 and directly receives image data from imaging platform I00. In these latter examples, phone 585 processes data to determine thelocations of indicators, and presents those indicators on the screen ofphone 585.

In various examples, receiver 130 identifies targets in image datareceived from image-capture device 110. These targets are identifiedusing location information stored by the targets themselves (via tag542) in database 599. In prior schemes, users disable location functionson their smartphones to increase their privacy. In various inventiveaspects described herein, users enable those location functions andreport their locations regularly to database 599 to increase theirprivacy. In various examples, users enable location functions and reporttheir locations to be noticed, e.g., to provide an increased sense ofpersonal safety. Aspects described herein permit users to select whetherthey want to be “noticed” and tracked by a drone or want to beunobservable or unobserved by drones. Each has respective benefits andapplications. In various aspects, drone imagery provides a visualconfirmation of who a tracked phone or other transponder is with, whattype of vehicle the phone is being transported in, or othervisually-perceptible data related to the location or circumstances ofthe transponder.

FIG. 15 shows a flowchart illustrating exemplary methods of providingobscurant data. The steps can be performed in any order except whenotherwise specified, or when data from an earlier step is used in alater step. In at least one example, processing begins with step 1510,or with step 1532. For clarity of explanation, reference is herein madeto various components shown in FIGS. 1-14 that can carry out orparticipate in the steps of the exemplary method. It should be noted,however, that other components can be used; that is, exemplary method(s)shown in FIG. 15 are not limited to being carried out by the identifiedcomponents.

Step 1510 includes receiving image data including an image of a target,e.g., as discussed above with reference to image-capture device 110.Targets can include people, buildings, or other features discussedabove.

Step 1520 includes receiving a preference setting corresponding to thetarget, e.g., from database 599 or directly from tag 342, 542, or 543 asdiscussed above or Step 1520 or other steps shown in FIG. 15 can alsoinclude transmitting to database 599 a timestamp corresponding to theimage data, e.g., for logging the times of drone observation asdiscussed above.

Step 1530 includes determining obscurant data of at least a portion ofthe image data corresponding to the image of the target using thereceived preference setting and optionally the received image data. Thiscan be done as discussed herein. Obscurant data can include datadefining highlights or indicators as well as graphic structures toprotect privacy. In various aspects, the target is selected from thegroup consisting of a person, a building, or an animal. In variousaspects, the determining step 1530 including determining a size of theportion of the image data using the received image data, e.g., bydetecting image pixels corresponding to the target using, e.g., face- orshape-recognition algorithms.

In various aspects, step 1540 includes selectively modifying thereceived image data according to the determined obscurant data toprovide a surveillance image. The modification (e.g., applyingobscurants or indicators) can be performed, e.g., by processor 186 orreceiver 130. In various examples, step 1530 includes determining atleast a portion of the image data to be obscured in response to apreference setting requesting privacy and step 1540 includes obscuringthe determined at least a portion of the image data. In variousexamples, step 1530 includes determining at least a portion of the imageto be indicated in response to a preference setting requesting increasedvisibility of the target and step 1540 includes modifying the determinedat least a portion of the image data to include data of a visibleindicator.

In various aspects, step 1532 includes receiving a unique identifier ofthe target, e.g., a GUID. Step 1520 includes step 1534 of transmittingthe received unique identifier to a database and step 1536 of receivingfrom the database the preference setting corresponding to the uniqueidentifier.

In various aspects, the image data correspond to a visibility region, asdiscussed above. Step 1520 includes step 1538 of transmitting data(e.g., a visibility polygon) of the visibility region to a database todetermine whether the target is present in the visibility region. Step1536 includes receiving from the database the preference setting or anindication that the target is not present in the visibility region. Ifthe target is not present, steps 1530 and 1540 can be omitted.

Various aspects include step 1538 or storing the determined obscurantdata in a storage device. The storage device can be a tamper-evidentstorage device, storage 131 with security unit 1056.

FIG. 16 shows a flowchart illustrating exemplary methods of providingsurveillance image data. As discussed above, the order of steps shownand components identified are not limiting. In various examples,processing begins with one of steps 1602, 1610, 1631, or 1632.

Step 1610 includes capturing image data including an image of a target.Step 1620 includes querying database 599 to receive a preference settingcorresponding to the target. Step 1630 includes determining obscurantdata of at least a portion of the image data corresponding to the targetor the image of the target using the received preference setting andoptionally the received image data, e.g., as discussed herein. Step 1640includes selectively modifying the received image data according to thedetermined obscurant data to provide the surveillance image data. Thiscan be performed as discussed above.

In various aspects, the image data correspond to a visibility region.Querying step 1620 includes decision step 1625 of querying the databasewith the visibility region to determine whether a target is present inthe visibility region. If the target is present, the determining andmodifying steps 1630, 1640 are performed. Querying step 1620 can includestep 1638 of providing to the database coordinates of a visibilitypolygon corresponding to the visibility region. This can be performed asdiscussed above.

In various aspects, querying step 1620 includes receiving from thedatabase data a masking layer representing one or more area(s) to bemasked, e.g., as discussed above with reference to FIGS. 6-8.Determining step 1630 includes determining coordinates in the image datacorresponding to the area(s).

In various aspects, steps 1602 and 1604 precede step 1620. In step 1602,the preference setting and an identity of the target are received. Instep 1604, the received setting and identity are stored in database 599.This can be done as discussed above, e.g., by providing a Web ortelephone interface a user can employ to register a preference regardingobscuring or indicating.

In various aspects, step 1632 includes receiving a unique identifiercorresponding to the target, e.g., from sensor 120, a location provider(e.g., a cellular telephone service provider), or analysis of the imagedata of the tag. In various aspects, step 1632 includes transmitting aradio-frequency (RF) interrogation signal and receiving an RFidentification signal in response, the RF identification signalincluding data of the unique identifier. Querying step 1620 includesstep 1634 of transmitting the unique identifier to the database.

Various aspects of the image analysis described in the previousparagraph include locating a tag of the target in the image data, e.g.,as discussed above, and decoding a target identifier of the tag. Theseaspects are useful with tags configured so that the target identifier isvisually represented in the image data. Querying step 1620 includestransmitting the decoded target identifier to the database 599. In otheraspects, the barcode indicates a preference setting or a group of peoplesharing a common preference setting. In various aspects describedherein, a unique identifier can identify a group of people or targets.The barcode or other visual representation can include instructions tothe drone. In an example, the information in the representationcorresponds to a policy setting stored in database 599, such as avalidity period. For example, a barcode with the data “42” cancorrespond to a record indicating that privacy from drones is permittedduring certain hours of the day or certain days of the year, or incertain locations. The number “42” can be, e.g., worn as a barcode by anumber of people. The data “42” can be encrypted, signed, or both. “42”can be a database key. All people wearing the same code are treated thesame way by the imaging platform 100. Tags can also correspond tosecurity keys. In an example, police cars have specific tags for eachday. The data in a barcode can be cryptographically signed toauthenticate it against the database 599 or another central authority.Digital signatures can be stored and transmitted as part of the uniqueID or can be stored or transmitted separately. All the aspects describedin this paragraph apply to non-visible tag IDs such as RF-transmittedgroup or policy IDs. All the aspects described in this paragraph applyto individuals as well as groups. Group codes advantageously retain someanonymity of individuals even though the group membership can betracked. Group membership can be recorded with preference settings orseparately from preference settings. Validity periods and cryptographicsignatures can be combined, e.g., to ensure that validity periods arenot forged. Data indicating the validity period or other policysettings, and cryptographic signatures corresponding to those settings,can be stored on the tag 542, in database 599, or in other data storagesystems 940. For example, the tag can include cryptographically-signeddata of its validity period. Revocation lists as known in the art can beused to indicate that a selected tag or unique ID should not beconsidered valid even if that ID is correctly signed. In variousexamples, the identifier can be associated with validation information,e.g., validity period, signature, or issuing authority. Methodsdescribed herein, e.g., FIGS. 15 and 16, can further include receivingan identifier of the target and validation information of the identifierand determining whether a use of the identifier satisfies the validationinformation, e.g., by testing the date or signature as described above.

In various examples, an image-capture system provides image data to aprocessor. The processor determines obscurant data and provides theobscurant data to the image-capture system. The image-capture systemthen modifies the image data according to the obscurant data to providethe surveillance image data.

In various examples, a copy of the data in database 599 is uploaded intoa data storage system 940 in the imaging platform 100. This permits theimaging platform 100, in various examples, to operate withouttransmitting any RF signals that might alert a criminal to the presenceof a surveillance device. The imaging platform 100 can operateautonomously and internally record image data and obscurant data forlater retrieval.

In various examples, the database 599 pushes data to imaging platform100 (or receiver 130, and likewise throughout this paragraph). Imagingplatform 100 can register with database 599 to receive updates relevantto, e.g., a certain geographical area (or target type, e.g., people orhouses; or unique-ID range or another feature; or all data known to thedatabase), and database 599 can transmit unique IDs, coordinates,obscurant data, preference settings, or other information to imagingplatform 100, e.g., as targets move in and out of the geographical area(or other registered category). Database 599 can provide updates ingroups of one or more records per transmission. Database 599 can pushdata, or imaging platform 100 can pull data, on a schedule or on demand.

In various examples, imaging platform 100 includes a security system,e.g., for an office, mall, or other space with controlled access to oneor more area(s). Employees or other authorized persons can be issueddo-not-track tags and targets can be recorded if not tagged, or visitorsor other persons not authorized can be issued do-track tags and targetsnot recorded unless tagged. In either situation (or other combinations)the privacy of authorized persons is protected and the activities ofothers (e.g., members of the general public) are recorded.

In various examples discussed above, the tag has a visible barcode orother visual representation of a unique ID. Imaging platform 100 (orreceiver 130) can recognize the barcode in the image data, decode thebarcode to determine the unique ID, and look up the unique ID indatabase 599 or perform other processing described above.

A surveillance system can comprise a storage system storing anidentifier of a target and a corresponding preference setting; animage-capture device that produces image data of a scene; a targetingdevice for locating a target in the image data and, if a target islocated, providing an identifier of the located target; and a processorthat receives the image data from the image-capture device and theprovided identifier from the targeting device, the processor configuredto retrieve from the storage system the preference setting correspondingto the provided identifier and selectively modify at least a portion ofthe image data corresponding to the located target based on thepreference setting.

The targeting device can include the processor. The system can include adisplay configured to receive and display the modified image data fromthe processor. The processor can be configured to obscure the at least aportion of the image data in response to a preference setting requestingprivacy. The system can include a user interface or other device forreceiving the preference setting from a user and storing the identifierof the target and the received preference setting in the storage system.The user interface can include a processor configured to transmit theidentifier of the target and the received preference setting to thestorage system via a network.

A tag can be configured to provide the identifier of the located target.The tag can include a visible indication of the identifier of thelocated target, and the targeting device can be configured to locate thevisible indication in the image data and determine the identifier of thelocated target using the image data corresponding to the visibleindication. The targeting device can be configured to locate specificcolors, shapes, or other visual features in the image data of a frame.The tag can include a transponder configured to provide aradio-frequency (RF) identification signal of the identifier of thelocated target, and the targeting device can include an RF receiverconfigured to receive the RF identification signal. The targeting devicecan include transmission electronics configured to provide an RFinterrogation signal to the tag, the tag configured to transmit the RFidentification signal in response to the RF interrogation signal. Thetargeting device can be configured to determine at least a portion ofthe image data corresponding to the located target by measuring the RFidentification signal and detecting a location of the tag with referenceto the image-capture device using the measurements.

The system can include a location provider configured to providelocation data of the tag. The storage system can be further configuredto store the location data in association with the identifier of thetarget. The image-capture device can be configured to provide visibilitydata of a spatial region corresponding to the scene, e.g., a visibilitypolygon. The targeting device can be configured to provide thevisibility data to the storage system and retrieve from the storagesystem the identifier of the target if the target can be within thespatial region.

In various examples, the spot can be indicated by the stored locationinstead of using the image data. For example, the targeting device canbe further configured to retrieve the stored location data of the targetif the target is within the spatial region. The processor can be furtherconfigured to determine the at least a portion of the image data usingthe retrieved location data and the visibility data, e.g., to map thecoordinates onto the visibility polygon and project onto the image as isknown in the 3-D graphics-processing art.

The tag can include the location provider and be configured to transmitthe location data and the identifier of the located target to thestorage system. The tag can include a cellular telephone, e.g., runningan app. The tag can include a transponder configured to provide aradio-frequency (RF) identification signal of the identifier of thelocated target, and the location provider can include one or more RFreceiver(s) configured to receive the RF identification signal anddetermine the location of the tag. For example, if the tag includes amobile telephone or other wireless communications terminal, a providerof network connectivity for the terminal can use triangulation or othertechniques to spatially locate the terminal. The target can be a personor a building.

In various examples, obscuring or indicated are performed on the imagingplatform 100. The platform includes an optoelectronic image-capturedevice 110 that produces image data of a scene and visibility data of aspatial region corresponding to the scene, e.g., a visibility polygon. Atag sensor 120 is configured to detect one or more tag(s) in the spatialregion and to detect respective unique identifier(s) (UIDs) of thedetected tag(s), A communications interface is configured to communicatewith a storage system. A processor is adapted to receive the image dataand the UID(s), retrieve respective preference setting(s) for the UID(s)from the storage system via the communications interface, andselectively modify portion(s) of the image data corresponding to thedetected tag(s) in response to the respective preference setting(s).

In various examples, obscuring or indicating are performed usinginformation provided by the platform 100. In some examples, the imagingplatform 100 include an optoelectronic image-capture device 110 thatproduces image data of a scene and visibility data of a spatial regioncorresponding to the scene, e.g., a visibility polygon. A tag sensor 120detects one or more tag(s) in the spatial region and detects respectiveunique identifier(s) (UIDs) of the detected tag(s). A communicationsinterface is configured to provide the image data and the UlDs of thetags to a receiver. The tag sensor can be further configured to detectrespective location(s) of the tag(s) and the communications interfacecan further be configured to provide the location(s) to the receiver.

In various examples, an imaging system for providing an image caninclude a database 599 storing information about whether to modify imagedata corresponding to a target. An optoelectronic image-capture devicecan produce image data. A processor can receive the image data andprovides the image. A targeting device can determine that a target ispresent in a frame of image data captured by the image-capture device.This can be done by, e.g., face recognition or feature recognition, orother techniques described above. The processor in some examples isresponsive to the targeting device to, when the target is present in theframe, query the database to determine whether to modify the image datafor a particular target. The database provides a query responseincluding a preference setting when the target is present, and theprocessor modifies the image data only if the preference settingindicates such modification should be performed. (Note that obscuringsome data automatically highlights the non-obscured data, which anobserver might be interested in, as discussed above.) The processorselectively modifies at least a portion of the image data of that framecorresponding to the detected target based on the query result.

Various combinations of the above can be used. For example, determininga query for the database, performing the database query, producingobscurant data, and modifying the image according to the query resultsand the detected target(s) can be performed in any order, subject toconstraints resulting from one operation providing data that is input toanother operation. The steps or portions of the steps can be performedby any components of imaging system 190, as discussed above. Data can bepassed both ways between the components, e.g., back and forth betweenthe platform 100 and the receiver 130 or processors (e.g., processor186) in those components. The processor 186 can query the database andprovide the preference setting to the receiver 130. The processor 186can provide obscurant data. The processor 186 can detect tags andprovide identifiers to the receiver 130, which can query the database.The processor 186 can provide no information beyond image data to thereceiver 130. In various examples, the image captured by the imagingplatform 100 includes image data of known objects or landmarks, and thereceiver 130 locates those image data, determines the position of theimaging platform 100 at the time of capture by projecting the image dataonto the known coordinates of the landmarks, and determines a visibilitypolygon from the determined position or the positions of the landmarks.In various examples, the image from the imaging platform 100 includesimages of tags, e.g., tag 342, and the receiver 130 locates the tags inthe images, determines the unique identifiers, and queries the database.

In various examples, sensor 120 produces a sensor image having a knownrelationship with the image data (e.g., of a human-visible image)captured by image-capture device 110. For example, sensor 120 caninclude an infrared (IR) camera or a directional antenna, and the sensorimage can represent intensity of IR or RF emission, respectively. Theprocessor 186 or the receiver 130 can determine the locations oridentifications of tags using the sensor image and then apply obscurantsto the image data. Conventional image-processing techniques such asprojection and affine transforms can be used to map pixel locations inthe sensor image to pixel locations in the image data.

In various example, image-capture device 110 produces a false-colorimage, e.g., of infrared or ultraviolet light, X-rays, magneticresonance images, computed tomography images, or other radiation orenergy. The image data thus represent a view of one or more target(s)other than, or in addition to, a visible-light view. Obscurants orindicators can be added to this image as described above for a visibleimage.

In various examples, a surveillance device or system is connected via awired or wireless link to a monitor console. The surveillance devicetransmits image data to the monitor console. The surveillance device canalso transmit location data, e.g., which way the camera on thesurveillance device is pointing, or what the camera is presentlyviewing. The surveillance device can also transmit data of parts of thescene (e.g., latitude and longitude, or references to stationary orother features visible in the image, e.g., fiducials, buildings, orcolumns within a building) or parts of the image (e.g., x and ycoordinates) that should be represented on the monitor console with avaried graphic appearance, e.g., superimposed with a feature to render asubject visible in the image data more or less visible. The surveillancedevice can also transmit image data having such variations applied. Themonitor console can alternatively perform this processing, or themonitor console and surveillance device can share the processing.

The monitor console or surveillance device can locate subjects ofinterest or potential interest in the image data or field of view of thecamera. Alternatively, a separate locator device, circuit, or programcan locate subjects of interest or potential interest.

In various examples, the surveillance device or system is an example ofan imaging platform I00 or imaging system 190. The monitor console is anexample of a receiver. The varied graphic appearance is an example of anobscurant or highlight. The subject is an example of a target.

The invention is inclusive of combinations of the aspects describedherein. References to “a particular aspect” and the like refer tofeatures that are present in at least one aspect of the invention.Separate references to “an aspect” or “particular aspects” or the likedo not necessarily refer to the same aspect or aspects; however, suchaspects are not mutually exclusive, unless so indicated or as arereadily apparent to one of skill in the art. The use of singular orplural in referring to “method” or “methods” and the like is notlimiting. The word “or” is used in this disclosure in a non-exclusivesense, unless otherwise explicitly noted.

In another embodiment the invention is a surveillance drone system wherethe drone carries the means for analyzing image data it is acquiring andapplying filters to obscure the identity of individuals who are not theapproved targets of surveillance, or in the alternative to obscure imagedata for areas outside of approved target areas of surveillance. Incases where neither an individual target nor an area target is withinthe surveillance the drone system would not transmit or recordsurveillance data. In such cases the only transmission would be of imagedata for the purposes of remotely piloting the drone.

In another embodiment the system would not transmit surveillance data,and would only store obscured surveillance date on the drone. Review ofthe surveillance would require data transfer after the drone hadcompleted flight, and then only the obscured data would be available.

The invention has been described in detail with particular reference tocertain preferred aspects thereof, but it will be understood thatvariations, combinations, and modifications can be effected by a personof ordinary skill in the art within the spirit and scope of theinvention.

What is claimed is:
 1. A surveillance system, comprising one or morehardware computing devices in electronic communication with each of asurveillance device, a display device, and a database storing preferencesettings that describe desired modifications of a video stream, the oneor more hardware computing devices storing computer program instructionsthat, upon execution, cause the one or more hardware computing devicesto: receive a surveillance video stream comprising a plurality of framesarranged in a sequence and captured, for a duration from a first time toa second time, by the surveillance device, the surveillance video streamrepresenting a surveillance area visible to the surveillance device forthe duration; identify, in the surveillance video stream, surveillancedata indicating that one or more objects present in the surveillancearea for the duration each bear a first tag; determine from thesurveillance data a first identifier derived from the first tag; usingthe first identifier, obtain from the database a first set of preferencesettings associated with the first tag; obtain modification dataidentified using the first set of preference settings; transform eachframe of the plurality of frames using the modification data to producea modified surveillance video stream in which a first object of the oneor more objects is displayed according to the first set of preferencesettings; and send the modified surveillance video stream to the displaydevice.
 2. The system of claim 1, wherein the computer programinstructions, upon execution, further cause the one or more hardwarecomputing devices to: determine that the first set of preferencesettings is associated with a first user of a plurality of usersauthorized to enter information into the database; and beforetransforming each of the plurality of frames, determine that the firstobject is associated with the first user.
 3. The system of claim 2,wherein the computer program instructions, upon execution, further causethe one or more hardware computing devices to: generate a user interfaceenabling the first user to enter the first set of preference settingsinto the user interface in order to control, in the modifiedsurveillance video stream, an appearance of the first object bearing thefirst tag as obscured, highlighted, or unchanged from the surveillancevideo stream; and send the user interface to a user interface system fordisplay to the first user.
 4. The system of claim 3, wherein the userinterface further enables the first user to enter the first set ofpreference settings to selectively change the first tag between ado-not-track tag and a please-track tag, and to obtain the modificationdata, the one or more hardware computing devices: responsive to adetermination that the first tag is a do-not-track tag, identifyobscurant data as the modification data; and responsive to adetermination that the first tag is a please-track tag, identify ahighlighting indicator as the modification data.
 5. The system of claim1, wherein the computer program instructions, upon execution, furthercause the one or more hardware computing devices to determine a validityperiod and to transform the plurality of frames to produce the modifiedsurveillance video stream only if the first time is within the validityperiod.
 6. The system of claim 1, wherein the surveillance device is animaging platform disposed within a drone, and the one or more hardwarecomputing devices include a processor and memory both disposed withinthe drone, the memory storing the instructions and the processorexecuting the instructions.
 7. The system of claim 1, wherein thecomputer program instructions, upon execution, further cause the one ormore hardware computing devices to: identify, based on informationstored in memory of the one or more hardware computing devices, thefirst tag as a do-not-track tag; select obscurant data as themodification data associated with the first object; and to transformeach frame of the plurality of frames of the surveillance video stream,apply the obscurant data to the frame to cause the modified surveillancevideo stream to include the first object obscured.
 8. The system ofclaim 7, wherein the computer program instructions, upon execution,further cause the one or more hardware computing devices to: determine,based at least in part on the surveillance video stream, that a secondtag different from the first tag is present in the surveillance area atthe first time; determine, based on the first set of preferencesettings, that targets associated with the second tag are to behighlighted; identify in the surveillance data a second objectassociated with the second tag; select a highlighting indicator as themodification data associated with the second object; and to transformeach frame of the plurality of frames of the surveillance video stream,further add a highlighting indicator to the frame to cause the modifiedsurveillance video stream to include increased visibility of the secondobject.
 9. The system of claim 8, wherein to transform the image data,the one or more hardware computing devices execute the computer programinstructions to further: determine that the first object is at a firstlocation in the surveillance area and depicted in a first portion of theimage data at the first time, and the second object is at a secondlocation in the surveillance area and depicted in a second portion ofthe image data at the first time; determine that the first portion ofthe image data and the second portion of the image data overlap at anoverlapping portion; determine which of the first location and thesecond location is closer to the surveillance device; responsive to adetermination that the first location is closer to the surveillancedevice than the second location, apply the obscurant data to the firstportion including the overlapping portion, and add the highlightingindicator to the second portion excluding the overlapping portion; andresponsive to a determination that the second location is closer to thesurveillance device than the first location, apply the obscurant data tothe first portion excluding the overlapping portion, and add thehighlighting indicator to the second portion including the overlappingportion.
 10. The system of claim 7, wherein to transform the image data,the one or more hardware computing devices execute the computer programinstructions to further: determine that the first object is at a firstlocation in the surveillance area and is depicted in a first portion ofthe image data at the first time; determine that the first portion ofthe image data overlaps, at an overlapping portion, a second portion ofthe image data depicting a second object present at a second location inthe surveillance area at the first time; determine that the secondlocation is closer than the first location to the surveillance device;and apply the obscurant data to the first portion excluding theoverlapping portion, to produce the modified surveillance video streamwith the first object substantially obscured and the second object notobscured.
 11. The system of claim 1, wherein the computer programinstructions, upon execution, further cause the one or more hardwarecomputing devices to: identify, based on information stored in memory ofthe one or more hardware computing devices, the first tag as aplease-track tag; select a highlighting indicator as the modificationdata associated with the first object; and to transform each frame ofthe plurality of frames of the surveillance video stream, apply thehighlighting indicator to the frame to cause the modified surveillancevideo stream to include the first object with increased visibilityrelative to the surveillance video stream.
 12. A surveillance dataprocessing system comprising one or more hardware computing devicesstoring computer program instructions that, upon execution, cause theone or more hardware computing devices to: receive a video streamcomprising image data captured at a first time by a surveillance device,the image data representing a surveillance area visible to thesurveillance device at the first time; determine, based at least in parton the image data, that a first tag is present in the surveillance areaat the first time, wherein the first tag is not a unique identifier ofany specific object present in the surveillance area; identify a firsttarget appearing in the image data, the first target representing afirst object bearing the first tag; identify a first portion of theimage data that is associated with the first target; determine that thefirst object is at a first location in the surveillance area at thefirst time; determine that the first portion of the image data overlaps,at an overlapping portion, a second portion of the image data depictinga second object present at a second location in the surveillance area atthe first time; determine that the second location is closer than thefirst location to the surveillance device; transform a target portion ofthe image data, the target portion comprising the first portionexcluding the overlapping portion, using modification data associatedwith the first tag to produce surveillance image data; and send amodified surveillance video stream comprising the surveillance imagedata to a display device in communication with the one or more hardwarecomputing devices.
 13. The system of claim 12, wherein the computerprogram instructions, upon execution, further cause the one or morehardware computing devices to: identify, based on information stored inmemory of the one or more hardware computing devices, the first tag as ado-not-track tag; and to transform the target portion of the image data:select obscurant data as the modification data; and apply the obscurantdata to the target portion of the image data to produce the surveillanceimage data comprising an image of the surveillance area at the firsttime wherein the first object is substantially obscured and the secondobject is not obscured.
 14. The system of claim 13, wherein the computerprogram instructions, upon execution, further cause the one or morehardware computing devices to: determine, based at least in part on theimage data, that a second tag different from the first tag is present inthe surveillance area at the first time; determine that the second tagis associated with a first preference setting indicating that targetsassociated with the second tag are to be highlighted in the modifiedsurveillance video stream; identify a second target appearing in theimage data, the second target representing the second object andassociating the second object with the second tag; identify a secondportion of the image data that is associated with the second target; andtogether with transforming the target portion of the image data, add ahighlighting indicator to the second portion of the image data toproduce the surveillance image data, such that in the modifiedsurveillance video stream the first object is obscured and visibility ofthe second object is increased.
 15. The system of claim 14, furthercomprising a database accessible by the one or more hardware computingdevices and storing a plurality of records each including: acorresponding unique identifier of a plurality of unique identifiers;and one or more preference settings describing a desired manipulation ofimage data; wherein the computer program instructions, upon execution,further cause the one or more hardware computing devices to: beforereceiving the video stream: generate a user interface enabling a user toenter the corresponding one or more preference settings for one or moreof the plurality of records; send the user interface to a user interfacesystem for display to the user; receive, from the user interface system,user input comprising a first set of preference settings including thefirst preference setting; and store the first set of preference settingswith a first unique identifier of the plurality of unique identifiers ina first record of the plurality of records; and to determine that thesecond tag is associated with the first preference setting: determinethat the second tag includes the first unique identifier; and query thedatabase using the first unique identifier to retrieve the first set ofpreference settings from the first record.
 16. The system of claim 14,wherein the computer program instructions, upon execution, further causethe one or more hardware computing devices to: receive sensor datacaptured at the first time by a sensor of the surveillance device, thesensor being configured to detect a signal emitted by a configurable tagpresent in the surveillance area; determine that the sensor dataincludes the signal comprising a first flag; determine whether the firstflag indicates the configurable tag is configured as a do-not-track tagor as a please-track tag; responsive to a determination that theconfigurable tag is configured as a do-not-track tag, execute theinstructions using the configurable tag as the first tag; and responsiveto a determination that the configurable tag is configured as aplease-track tag, execute the instructions using the configurable tag asthe second tag.
 17. The system of claim 12, wherein the computer programinstructions, upon execution, further cause the one or more hardwarecomputing devices to: identify, based on information stored in memory ofthe one or more hardware computing devices, the first tag as aplease-track tag; and to transform the target portion of the image data:select a highlighting indicator as the modification data; and apply thehighlighting indicator to the target portion of the image data toproduce the surveillance image data comprising an image of thesurveillance area at the first time wherein visibility of the firstobject is increased.
 18. The system of claim 12, wherein the computerprogram instructions, upon execution, further cause the one or morehardware computing devices to: receive sensor data captured at the firsttime by a sensor of the surveillance device; and determine, based on thesensor data, that the first tag is present in the surveillance area atthe first time, the first tag emitting a signal that is detectable bythe sensor.
 19. The system of claim 18, wherein the computer programinstructions, upon execution, further cause the one or more hardwarecomputing devices to: determine that the sensor data includes a firstvalue of a first flag, the first value set by a switch on the first tag;and before transforming the target portion of the image data, determinethat the first value indicates the first object bearing the first tag isto be obscured in the surveillance image data.
 20. The system of claim12, wherein the surveillance device is a drone having an imaging deviceand the one or more hardware computing devices include a processor andmemory both disposed within the drone, the memory storing theinstructions and the processor executing the instructions.